nit: please add a gauge metric for slow activities count with activity name as label. I think this will be useful for folks who rely on metrics to actively monitor etcd cluster.

I think we need to decide the naming better.

a) ```
endActivity := wd.StartActivity("saving v2 snapshot")
endActivity()

b)  ```
  markAsDone := wd.Start("saving v2 snapshot")
  markAsDone()

c) ```
markAsDone := wd.Notify("saving v2 snapshot")
markAsDone()

d) ```
 endScope := wd.StartScope("saving v2 snapshot")
 endScope()

We might also have a synthactic sugar:

or watchdog.executeInScope("saving v2 snapshot", func() { ...} );

I don't like 'cancel' as it suggests an interruption of an existing process.. and it's not.

I hoped we will settle this (API / naming) in a document rather than keep updating the PR, that is relatively expensive.

I like the suggestion of synthactic sugar. I imagine it's similar to bboltDB's View and Update method

StartXXX seems not good to me, it doesn't start any activity, instead essentially it just registers the activity to watchdog. So how about Register and unregister?

unregister := wd.Register("saving v2 snapshot")
unregister()

Users can also call the syntactic sugar as you proposed below,

wd.Execute("saving v2 snapshot", fn)

Please also see the updated doc https://docs.google.com/document/d/1U9hAcZQp3Y36q_JFiw2VBJXVAo2dK2a-8Rsbqv3GgDo/edit#heading=h.3oeryohw1c9o

I wonder what do you think about pre-registering type of activities that are tracked in the watchdog:

{ at the package level:
  var SAVING_V2_SNAPSHOTS = watchdog.RegisterActivity("saving v2 snapshots");
}```


endActivity := wd.StartActivity(SAVING_V2_SNAPSHOTS)
endActivity()

The main benefit is that it drives the API usage to small number of strings rather than patterns like:

wd.StartActivity(string.format("Writing log entry: %d", entryId));

thus the API can be e.g. used for monitoring how long classes of activities take in scope of a watchdog.
i.e. watchdog sees that 17% of wall-time it was in the writing-logs routine. 

It's a stretch and over design. but in theory such registration allows to build also a hierarchy of 'activities/scope'. 

The fundamental difference between this design and watchdog pattern in general is that you propose to monitor a specific registered activities.

I assumed the goal is to monitor watchdog as a whole thing....

Watchdog is unhealthy when for the whole 'timeout' it got no 'Start' nor 'Stop/Done' notifications.
Any such notifications does reset the watchdog. Lack of new activities is especially an alart.

We do register specific activities only to:
a) have actionable error messages:
- the watchdog is stuck with following active activities...
- the watchdog is stuck and the last started / DONE activity is X
b) [stretch] We might piggy back on it and collect metrics what percent of time specifica activities are active state